190 PART 4 Comparing Groups

So, how do you interpret a κ of 0.5138? There’s no universal agreement as to an

acceptable value for . One common convention is that values of κ less than 0.4 are

considered poor, those between 0.4 and 0.75 are acceptable, and those more than

0.75 are excellent. In this case, our raters may be performing acceptably.

For CIs forκ, you won’t find an easy formula, but the fourfold table web page

(https://statpages.info/ctab2x2.html) provides approximate CIs. For the

preceding example, the 95 percent CI is 0.202 to 0.735. This means that for your

two raters, their agreement was 0.514 (95 percent CI 0.202 to 0.735), which sug-

gests that the agreement level was acceptable.

You can construct a similar table to Figure 13-6 for estimating intra-rater reli-

ability. You would do this by having one rater rate the same groups of scans in two

separate sessions. In this case, in the table in Figure 13-6, you’d replace the by

Rater with in Session in the row and column labels.